Repository navigation
Conversation
A full worker slept a fixed 1 s before taking again. With jobs queued a job-take costs ~0.5 s server-side whatever its size, so a 32-slot vLLM worker ran take (0.5 s) -> jobs (~0.4 s, inside the sleep) -> idle, one 32-job batch per 1.5 s. It now waits until no job is in progress, at most FULL_WAIT_SECONDS (the old sleep), so it takes the same or a bigger batch no later than before. Taking each freed slot at once was tried and is slower: smaller takes pay the same 0.5 s each.
There was a problem hiding this comment.
🟡 Changes recommended
An empty worker with zero concurrency now enters a tight loop that starves other event-loop tasks.
1 open issue
What changed in this PR
This PR improves saturated serverless worker throughput by replacing fixed polling delays with a bounded wait for the worker to drain, allowing earlier full-batch job acquisition.
Changes:
- Wake job acquisition when all slots are free, on shutdown, or after one second.
- Signal job completion through an asynchronous event.
- Add regression tests for full draining, partial draining, timeout, and shutdown.
| File | Description |
|---|---|
| tests/test_serverless/test_rp_scale.py | Adds coverage for bounded batch waits and shutdown behavior. |
| runpod/serverless/modules/rp_scale.py | Implements event-driven draining before fetching another batch. |
Review effort: Balanced
- _wait_until_drained returned immediately on an empty worker even when concurrency_modifier returned 0. set_scale also returns without awaiting in that state, so get_jobs spun without yielding and starved the event loop, including shutdown. An empty worker now only counts as drained when it has slots; otherwise it waits out FULL_WAIT_SECONDS or shutdown, as the old sleep did. - The scaling_up worker test stopped the worker by mocking is_alive without setting the shutdown event, so run_jobs could wait on it forever (hung on 3.10/3.11). The mock now calls kill_worker() when it starts returning False, as a real stop does.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a worker is at its concurrency limit,
JobScaler.get_jobsslept a fixed 1s before checking for room again. With jobs queued, a job-take costs ~0.5s on the platform side, whatever its size. So a saturated 32-slot vLLM worker ran this cycle:That is one 32-job batch every ~1.4s, with the GPU idle for most of each cycle.
FULL_WAIT_SECONDS = 1(the old sleep), then takes a full batch.handle_jobsets anasyncio.Eventwhen a job leaves the worker; the wait also ends on shutdown.Because the wait is bounded by the old sleep, a full worker takes the same or a bigger batch, no later than before. One long-running job can't hold the other slots idle for more than 1s, and
set_scale()still runs at least as often.Why not take each slot as soon as it frees: I prototyped that too, with and without a linger to batch. It was slower (512-job burst finished 99% of jobs 4–7s later). Every take pays the same ~0.5s, so smaller takes mean fewer jobs per round trip.
Impact
runpod/worker-v1-vllm:v2.28.0, Qwen2.5-0.5B-Instruct, RTX 4090,MAX_CONCURRENCY=32. One warm worker per arm, both arms loaded at the same time, 4 reps. The SDK is swapped via start command; the worker logs the patched file's sha256 to confirm which code ran./runsync, time to 99% donedelayTimeSequential requests and 32-job bursts (worker never saturated) are unchanged. Single-slot workers also take as soon as their job finishes, since occupancy 0 is all slots free.
The remaining limit is the ~0.5s job-take round trip itself, which is half of every cycle now. See #647.
Test plan
FULL_WAIT_SECONDStest_rp_scale.pyrepeated 5x, no flakesmake quality-check: 1540 passed, 91.91% coverage